haikuwebkit/Source/bmalloc/bmalloc/IsoDeallocator.h

55 lines
1.9 KiB
C
Raw Permalink Normal View History

bmalloc should support strictly type-segregated isolated heaps https://bugs.webkit.org/show_bug.cgi?id=178108 Reviewed by Saam Barati, Simon Fraser, and Ryosuke Niwa. Source/bmalloc: This introduces a new allocation API in bmalloc called IsoHeap. An IsoHeap is templatized by type and created in static storage. When unused, it takes only a few words. When you do use it, each IsoHeap gets a bag of virtual pages unique to it. This prevents use-after-free bugs in one IsoHeap from affecting any other memory. At worst, two pointers of the same type will point to the same object even though they should not have. IsoHeaps allocate using a first-fit discipline that combines ideas from bmalloc and Riptide (the JSC GC): Like Riptide, it uses a bump'n'pop allocator. What Riptide calls blocks, IsoHeaps calls pages. Pages are collected into directories. Directories track pages using bitvectors, so that it's easy to quickly find a completely free page or one that has at least one free object. I think that the bump'n'pop allocator is as fast as the bmalloc Immix-style (page and line) allocator, but is better at allocating in holes. It's guaranteed to follow a first-fit discipline. However, the real reason why I wrote it that was is that this is what I'm more familiar with. This is a part of the design I want to revisit (bug 179278). Like bmalloc, it uses a deallocation log. This means that the internal IsoHeap data structures can be locked with a coarse-grained lock, since the deallocator only grabs it when flushing the log. Similarly, the allocator only grabs it when refilling the bump'n'pop FreeList. This adds a unit test for IsoHeaps. In this change, IsoHeaps are adopted only by WebCore's RenderObject. Note that despite the use of GC concepts, it's not a goal to make this code directly sharable with GC. The GC will probably have to do isolated heaps its own way (likely a special Subspace or something like that). * bmalloc.xcodeproj/project.pbxproj: * bmalloc/Algorithm.h: (bmalloc::findBitInWord): * bmalloc/AllIsoHeaps.cpp: Added. (bmalloc::AllIsoHeaps::AllIsoHeaps): (bmalloc::AllIsoHeaps::add): (bmalloc::AllIsoHeaps::head): * bmalloc/AllIsoHeaps.h: Added. * bmalloc/AllIsoHeapsInlines.h: Added. (bmalloc::AllIsoHeaps::forEach): * bmalloc/BMalloced.h: Added. * bmalloc/Bits.h: Added. (bmalloc::bitsArrayLength): (bmalloc::BitsWordView::BitsWordView): (bmalloc::BitsWordView::numBits const): (bmalloc::BitsWordView::word const): (bmalloc::BitsWordOwner::BitsWordOwner): (bmalloc::BitsWordOwner::view const): (bmalloc::BitsWordOwner::operator=): (bmalloc::BitsWordOwner::setAll): (bmalloc::BitsWordOwner::clearAll): (bmalloc::BitsWordOwner::set): (bmalloc::BitsWordOwner::numBits const): (bmalloc::BitsWordOwner::arrayLength const): (bmalloc::BitsWordOwner::word const): (bmalloc::BitsWordOwner::word): (bmalloc::BitsWordOwner::words const): (bmalloc::BitsWordOwner::words): (bmalloc::BitsAndWords::BitsAndWords): (bmalloc::BitsAndWords::view const): (bmalloc::BitsAndWords::numBits const): (bmalloc::BitsAndWords::word const): (bmalloc::BitsOrWords::BitsOrWords): (bmalloc::BitsOrWords::view const): (bmalloc::BitsOrWords::numBits const): (bmalloc::BitsOrWords::word const): (bmalloc::BitsNotWords::BitsNotWords): (bmalloc::BitsNotWords::view const): (bmalloc::BitsNotWords::numBits const): (bmalloc::BitsNotWords::word const): (bmalloc::BitsImpl::BitsImpl): (bmalloc::BitsImpl::numBits const): (bmalloc::BitsImpl::size const): (bmalloc::BitsImpl::arrayLength const): (bmalloc::BitsImpl::operator== const): (bmalloc::BitsImpl::operator!= const): (bmalloc::BitsImpl::at const): (bmalloc::BitsImpl::operator[] const): (bmalloc::BitsImpl::isEmpty const): (bmalloc::BitsImpl::operator& const): (bmalloc::BitsImpl::operator| const): (bmalloc::BitsImpl::operator~ const): (bmalloc::BitsImpl::forEachSetBit const): (bmalloc::BitsImpl::forEachClearBit const): (bmalloc::BitsImpl::forEachBit const): (bmalloc::BitsImpl::findBit const): (bmalloc::BitsImpl::findSetBit const): (bmalloc::BitsImpl::findClearBit const): (bmalloc::BitsImpl::wordView const): (bmalloc::BitsImpl::atImpl const): (bmalloc::Bits::Bits): (bmalloc::Bits::operator=): (bmalloc::Bits::resize): (bmalloc::Bits::setAll): (bmalloc::Bits::clearAll): (bmalloc::Bits::setAndCheck): (bmalloc::Bits::operator|=): (bmalloc::Bits::operator&=): (bmalloc::Bits::at const): (bmalloc::Bits::operator[] const): (bmalloc::Bits::BitReference::BitReference): (bmalloc::Bits::BitReference::operator bool const): (bmalloc::Bits::BitReference::operator=): (bmalloc::Bits::at): (bmalloc::Bits::operator[]): * bmalloc/CryptoRandom.cpp: Replaced with Source/bmalloc/bmalloc/CryptoRandom.cpp. (bmalloc::cryptoRandom): * bmalloc/CryptoRandom.h: Replaced with Source/bmalloc/bmalloc/CryptoRandom.h. * bmalloc/DeferredDecommit.h: Added. * bmalloc/DeferredDecommitInlines.h: Added. (bmalloc::DeferredDecommit::DeferredDecommit): * bmalloc/DeferredTrigger.h: Added. (bmalloc::DeferredTrigger::DeferredTrigger): * bmalloc/DeferredTriggerInlines.h: Added. (bmalloc::DeferredTrigger<trigger>::didBecome): (bmalloc::DeferredTrigger<trigger>::handleDeferral): * bmalloc/EligibilityResult.h: Added. (bmalloc::EligibilityResult::EligibilityResult): * bmalloc/EligibilityResultInlines.h: Added. (bmalloc::EligibilityResult<Config>::EligibilityResult): * bmalloc/FixedVector.h: * bmalloc/FreeList.cpp: Added. (bmalloc::FreeList::FreeList): (bmalloc::FreeList::~FreeList): (bmalloc::FreeList::clear): (bmalloc::FreeList::initializeList): (bmalloc::FreeList::initializeBump): (bmalloc::FreeList::contains const): * bmalloc/FreeList.h: Added. (bmalloc::FreeCell::scramble): (bmalloc::FreeCell::descramble): (bmalloc::FreeCell::setNext): (bmalloc::FreeCell::next const): (bmalloc::FreeList::allocationWillFail const): (bmalloc::FreeList::allocationWillSucceed const): (bmalloc::FreeList::originalSize const): (bmalloc::FreeList::head const): * bmalloc/FreeListInlines.h: Added. (bmalloc::FreeList::allocate): (bmalloc::FreeList::forEach const): * bmalloc/IsoAllocator.h: Added. * bmalloc/IsoAllocatorInlines.h: Added. (bmalloc::IsoAllocator<Config>::IsoAllocator): (bmalloc::IsoAllocator<Config>::~IsoAllocator): (bmalloc::IsoAllocator<Config>::allocate): (bmalloc::IsoAllocator<Config>::allocateSlow): (bmalloc::IsoAllocator<Config>::scavenge): * bmalloc/IsoConfig.h: Added. * bmalloc/IsoDeallocator.h: Added. * bmalloc/IsoDeallocatorInlines.h: Added. (bmalloc::IsoDeallocator<Config>::IsoDeallocator): (bmalloc::IsoDeallocator<Config>::~IsoDeallocator): (bmalloc::IsoDeallocator<Config>::deallocate): (bmalloc::IsoDeallocator<Config>::scavenge): * bmalloc/IsoDirectory.h: Added. (bmalloc::IsoDirectoryBaseBase::IsoDirectoryBaseBase): (bmalloc::IsoDirectoryBaseBase::~IsoDirectoryBaseBase): (bmalloc::IsoDirectoryBase::heap): * bmalloc/IsoDirectoryInlines.h: Added. (bmalloc::IsoDirectoryBase<Config>::IsoDirectoryBase): (bmalloc::passedNumPages>::IsoDirectory): (bmalloc::passedNumPages>::takeFirstEligible): (bmalloc::passedNumPages>::didBecome): (bmalloc::passedNumPages>::didDecommit): (bmalloc::passedNumPages>::scavenge): (bmalloc::passedNumPages>::forEachCommittedPage): * bmalloc/IsoDirectoryPage.h: Added. (bmalloc::IsoDirectoryPage::index const): * bmalloc/IsoDirectoryPageInlines.h: Added. (bmalloc::IsoDirectoryPage<Config>::IsoDirectoryPage): (bmalloc::IsoDirectoryPage<Config>::pageFor): * bmalloc/IsoHeap.h: Added. (bmalloc::api::IsoHeap::allocatorOffset): (bmalloc::api::IsoHeap::setAllocatorOffset): (bmalloc::api::IsoHeap::deallocatorOffset): (bmalloc::api::IsoHeap::setDeallocatorOffset): * bmalloc/IsoHeapImpl.cpp: Added. (bmalloc::IsoHeapImplBase::IsoHeapImplBase): (bmalloc::IsoHeapImplBase::~IsoHeapImplBase): (bmalloc::IsoHeapImplBase::scavengeNow): (bmalloc::IsoHeapImplBase::finishScavenging): * bmalloc/IsoHeapImpl.h: Added. * bmalloc/IsoHeapImplInlines.h: Added. (bmalloc::IsoHeapImpl<Config>::IsoHeapImpl): (bmalloc::IsoHeapImpl<Config>::takeFirstEligible): (bmalloc::IsoHeapImpl<Config>::didBecomeEligible): (bmalloc::IsoHeapImpl<Config>::scavenge): (bmalloc::IsoHeapImpl<Config>::allocatorOffset): (bmalloc::IsoHeapImpl<Config>::deallocatorOffset): (bmalloc::IsoHeapImpl<Config>::numLiveObjects): (bmalloc::IsoHeapImpl<Config>::numCommittedPages): (bmalloc::IsoHeapImpl<Config>::forEachDirectory): (bmalloc::IsoHeapImpl<Config>::forEachCommittedPage): (bmalloc::IsoHeapImpl<Config>::forEachLiveObject): * bmalloc/IsoHeapInlines.h: Added. (bmalloc::api::IsoHeap<Type>::allocate): (bmalloc::api::IsoHeap<Type>::tryAllocate): (bmalloc::api::IsoHeap<Type>::deallocate): (bmalloc::api::IsoHeap<Type>::scavenge): (bmalloc::api::IsoHeap<Type>::isInitialized): (bmalloc::api::IsoHeap<Type>::impl): * bmalloc/IsoPage.h: Added. (bmalloc::IsoPage::index const): (bmalloc::IsoPage::directory): (bmalloc::IsoPage::isInUseForAllocation const): (bmalloc::IsoPage::indexOfFirstObject): * bmalloc/IsoPageInlines.h: Added. (bmalloc::IsoPage<Config>::tryCreate): (bmalloc::IsoPage<Config>::IsoPage): (bmalloc::IsoPage<Config>::free): (bmalloc::IsoPage<Config>::startAllocating): (bmalloc::IsoPage<Config>::stopAllocating): (bmalloc::IsoPage<Config>::forEachLiveObject): * bmalloc/IsoPageTrigger.h: Added. * bmalloc/IsoTLS.cpp: Added. (bmalloc::IsoTLS::scavenge): (bmalloc::IsoTLS::IsoTLS): (bmalloc::IsoTLS::ensureEntries): (bmalloc::IsoTLS::destructor): (bmalloc::IsoTLS::sizeForCapacity): (bmalloc::IsoTLS::capacityForSize): (bmalloc::IsoTLS::size): (bmalloc::IsoTLS::forEachEntry): * bmalloc/IsoTLS.h: Added. * bmalloc/IsoTLSAllocatorEntry.h: Added. * bmalloc/IsoTLSAllocatorEntryInlines.h: Added. (bmalloc::IsoTLSAllocatorEntry<Config>::IsoTLSAllocatorEntry): (bmalloc::IsoTLSAllocatorEntry<Config>::~IsoTLSAllocatorEntry): (bmalloc::IsoTLSAllocatorEntry<Config>::construct): * bmalloc/IsoTLSDeallocatorEntry.h: Added. * bmalloc/IsoTLSDeallocatorEntryInlines.h: Added. (bmalloc::IsoTLSDeallocatorEntry<Config>::IsoTLSDeallocatorEntry): (bmalloc::IsoTLSDeallocatorEntry<Config>::~IsoTLSDeallocatorEntry): (bmalloc::IsoTLSDeallocatorEntry<Config>::construct): * bmalloc/IsoTLSEntry.cpp: Added. (bmalloc::IsoTLSEntry::IsoTLSEntry): (bmalloc::IsoTLSEntry::~IsoTLSEntry): * bmalloc/IsoTLSEntry.h: Added. (bmalloc::IsoTLSEntry::offset const): (bmalloc::IsoTLSEntry::alignment const): (bmalloc::IsoTLSEntry::size const): (bmalloc::IsoTLSEntry::extent const): * bmalloc/IsoTLSEntryInlines.h: Added. (bmalloc::IsoTLSEntry::walkUpToInclusive): (bmalloc::DefaultIsoTLSEntry<EntryType>::DefaultIsoTLSEntry): (bmalloc::DefaultIsoTLSEntry<EntryType>::~DefaultIsoTLSEntry): (bmalloc::DefaultIsoTLSEntry<EntryType>::move): (bmalloc::DefaultIsoTLSEntry<EntryType>::destruct): (bmalloc::DefaultIsoTLSEntry<EntryType>::scavenge): * bmalloc/IsoTLSInlines.h: Added. (bmalloc::IsoTLS::allocate): (bmalloc::IsoTLS::deallocate): (bmalloc::IsoTLS::scavenge): (bmalloc::IsoTLS::allocator): (bmalloc::IsoTLS::deallocator): (bmalloc::IsoTLS::get): (bmalloc::IsoTLS::set): (bmalloc::IsoTLS::ensureHeap): (bmalloc::IsoTLS::ensureHeapAndEntries): * bmalloc/IsoTLSLayout.cpp: Added. (bmalloc::IsoTLSLayout::IsoTLSLayout): (bmalloc::IsoTLSLayout::add): * bmalloc/IsoTLSLayout.h: Added. (bmalloc::IsoTLSLayout::head const): * bmalloc/PerHeapKind.h: * bmalloc/PerProcess.h: (bmalloc::PerProcess<T>::getFastCase): * bmalloc/Scavenger.cpp: (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: * bmalloc/bmalloc.h: (bmalloc::api::scavengeThisThread): * test: Added. * test/testbmalloc.cpp: Added. (hiddenTruthBecauseNoReturnIsStupid): (usage): (assertEmptyPointerSet): (assertHasObjects): (assertHasOnlyObjects): (assertClean): (testIsoSimple): (testIsoSimpleScavengeBeforeDealloc): (testIsoFlipFlopFragmentedPages): (testIsoFlipFlopFragmentedPagesScavengeInMiddle): (BisoMalloced::BisoMalloced): (testBisoMalloced): (BisoMallocedInline::BisoMallocedInline): (testBisoMallocedInline): (run): (main): Source/WebCore: No new tests because no new change in behavior. Though, the bmalloc change has a unit test. Adopting IsoHeap means dropping in macros in both the .h and .cpp file of each class that we opt in. It's not pretty, but it helps ensure speedy allocation since it means that we never have to do any kind of switch or dynamic lookup to find the right allocator for a type. This change is perf-neutral on MotionMark, PLT3, and membuster. * Sources.txt: * html/shadow/SliderThumbElement.cpp: * html/shadow/SliderThumbElement.h: * html/shadow/mac/ImageControlsButtonElementMac.cpp: * html/shadow/mac/ImageControlsRootElementMac.cpp: * rendering/RenderAttachment.cpp: * rendering/RenderAttachment.h: * rendering/RenderBlock.cpp: * rendering/RenderBlock.h: * rendering/RenderBlockFlow.cpp: * rendering/RenderBlockFlow.h: * rendering/RenderBox.cpp: * rendering/RenderBox.h: * rendering/RenderBoxModelObject.cpp: * rendering/RenderBoxModelObject.h: * rendering/RenderButton.cpp: * rendering/RenderButton.h: * rendering/RenderCombineText.cpp: * rendering/RenderCombineText.h: * rendering/RenderCounter.cpp: * rendering/RenderCounter.h: * rendering/RenderDeprecatedFlexibleBox.cpp: * rendering/RenderDeprecatedFlexibleBox.h: * rendering/RenderDetailsMarker.cpp: * rendering/RenderDetailsMarker.h: * rendering/RenderElement.cpp: * rendering/RenderElement.h: * rendering/RenderEmbeddedObject.cpp: * rendering/RenderEmbeddedObject.h: * rendering/RenderFileUploadControl.cpp: * rendering/RenderFileUploadControl.h: * rendering/RenderFlexibleBox.cpp: * rendering/RenderFlexibleBox.h: * rendering/RenderFragmentContainer.cpp: * rendering/RenderFragmentContainer.h: * rendering/RenderFragmentContainerSet.cpp: * rendering/RenderFragmentContainerSet.h: * rendering/RenderFragmentedFlow.cpp: * rendering/RenderFragmentedFlow.h: * rendering/RenderFrameBase.cpp: * rendering/RenderFrameBase.h: * rendering/RenderFrameSet.cpp: * rendering/RenderFrameSet.h: * rendering/RenderFullScreen.cpp: * rendering/RenderFullScreen.h: * rendering/RenderGrid.cpp: * rendering/RenderGrid.h: * rendering/RenderHTMLCanvas.cpp: * rendering/RenderHTMLCanvas.h: * rendering/RenderImage.cpp: * rendering/RenderImage.h: * rendering/RenderImageResourceStyleImage.cpp: * rendering/RenderImageResourceStyleImage.h: * rendering/RenderInline.cpp: * rendering/RenderInline.h: * rendering/RenderLayerModelObject.cpp: * rendering/RenderLayerModelObject.h: * rendering/RenderLineBreak.cpp: * rendering/RenderLineBreak.h: * rendering/RenderListBox.cpp: * rendering/RenderListBox.h: * rendering/RenderListItem.cpp: * rendering/RenderListItem.h: * rendering/RenderListMarker.cpp: * rendering/RenderListMarker.h: * rendering/RenderMedia.cpp: * rendering/RenderMedia.h: * rendering/RenderMediaControlElements.cpp: * rendering/RenderMediaControlElements.h: * rendering/RenderMenuList.cpp: * rendering/RenderMenuList.h: * rendering/RenderMeter.cpp: * rendering/RenderMeter.h: * rendering/RenderMultiColumnFlow.cpp: * rendering/RenderMultiColumnFlow.h: * rendering/RenderMultiColumnSet.cpp: * rendering/RenderMultiColumnSet.h: * rendering/RenderMultiColumnSpannerPlaceholder.cpp: * rendering/RenderMultiColumnSpannerPlaceholder.h: * rendering/RenderObject.cpp: * rendering/RenderObject.h: * rendering/RenderProgress.cpp: * rendering/RenderProgress.h: * rendering/RenderQuote.cpp: * rendering/RenderQuote.h: * rendering/RenderReplaced.cpp: * rendering/RenderReplaced.h: * rendering/RenderReplica.cpp: * rendering/RenderReplica.h: * rendering/RenderRuby.cpp: * rendering/RenderRuby.h: * rendering/RenderRubyBase.cpp: * rendering/RenderRubyBase.h: * rendering/RenderRubyRun.cpp: * rendering/RenderRubyRun.h: * rendering/RenderRubyText.cpp: * rendering/RenderRubyText.h: * rendering/RenderScrollbarPart.cpp: * rendering/RenderScrollbarPart.h: * rendering/RenderSearchField.cpp: * rendering/RenderSearchField.h: * rendering/RenderSlider.cpp: * rendering/RenderSlider.h: * rendering/RenderTable.cpp: * rendering/RenderTable.h: * rendering/RenderTableCaption.cpp: * rendering/RenderTableCaption.h: * rendering/RenderTableCell.cpp: * rendering/RenderTableCell.h: * rendering/RenderTableCol.cpp: * rendering/RenderTableCol.h: * rendering/RenderTableRow.cpp: * rendering/RenderTableRow.h: * rendering/RenderTableSection.cpp: * rendering/RenderTableSection.h: * rendering/RenderText.cpp: * rendering/RenderText.h: * rendering/RenderTextControl.cpp: * rendering/RenderTextControl.h: * rendering/RenderTextControlMultiLine.cpp: * rendering/RenderTextControlMultiLine.h: * rendering/RenderTextControlSingleLine.cpp: * rendering/RenderTextControlSingleLine.h: * rendering/RenderTextFragment.cpp: * rendering/RenderTextFragment.h: * rendering/RenderVTTCue.cpp: * rendering/RenderVTTCue.h: * rendering/RenderVideo.cpp: * rendering/RenderVideo.h: * rendering/RenderView.cpp: * rendering/RenderView.h: * rendering/RenderWidget.cpp: * rendering/RenderWidget.h: * rendering/mathml/RenderMathMLBlock.cpp: * rendering/mathml/RenderMathMLBlock.h: * rendering/mathml/RenderMathMLFenced.cpp: * rendering/mathml/RenderMathMLFenced.h: * rendering/mathml/RenderMathMLFencedOperator.cpp: * rendering/mathml/RenderMathMLFencedOperator.h: * rendering/mathml/RenderMathMLFraction.cpp: * rendering/mathml/RenderMathMLFraction.h: * rendering/mathml/RenderMathMLMath.cpp: * rendering/mathml/RenderMathMLMath.h: * rendering/mathml/RenderMathMLMenclose.cpp: * rendering/mathml/RenderMathMLMenclose.h: * rendering/mathml/RenderMathMLOperator.cpp: * rendering/mathml/RenderMathMLOperator.h: * rendering/mathml/RenderMathMLPadded.cpp: * rendering/mathml/RenderMathMLPadded.h: * rendering/mathml/RenderMathMLRoot.cpp: * rendering/mathml/RenderMathMLRoot.h: * rendering/mathml/RenderMathMLRow.cpp: * rendering/mathml/RenderMathMLRow.h: * rendering/mathml/RenderMathMLScripts.cpp: * rendering/mathml/RenderMathMLScripts.h: * rendering/mathml/RenderMathMLSpace.cpp: * rendering/mathml/RenderMathMLSpace.h: * rendering/mathml/RenderMathMLToken.cpp: * rendering/mathml/RenderMathMLToken.h: * rendering/mathml/RenderMathMLUnderOver.cpp: * rendering/mathml/RenderMathMLUnderOver.h: * rendering/svg/RenderSVGBlock.cpp: * rendering/svg/RenderSVGBlock.h: * rendering/svg/RenderSVGContainer.cpp: * rendering/svg/RenderSVGContainer.h: * rendering/svg/RenderSVGEllipse.cpp: * rendering/svg/RenderSVGEllipse.h: * rendering/svg/RenderSVGForeignObject.cpp: * rendering/svg/RenderSVGForeignObject.h: * rendering/svg/RenderSVGGradientStop.cpp: * rendering/svg/RenderSVGGradientStop.h: * rendering/svg/RenderSVGHiddenContainer.cpp: * rendering/svg/RenderSVGHiddenContainer.h: * rendering/svg/RenderSVGImage.cpp: * rendering/svg/RenderSVGImage.h: * rendering/svg/RenderSVGInline.cpp: * rendering/svg/RenderSVGInline.h: * rendering/svg/RenderSVGInlineText.cpp: * rendering/svg/RenderSVGInlineText.h: * rendering/svg/RenderSVGModelObject.cpp: * rendering/svg/RenderSVGModelObject.h: * rendering/svg/RenderSVGPath.cpp: * rendering/svg/RenderSVGPath.h: * rendering/svg/RenderSVGRect.cpp: * rendering/svg/RenderSVGRect.h: * rendering/svg/RenderSVGResourceClipper.cpp: * rendering/svg/RenderSVGResourceClipper.h: * rendering/svg/RenderSVGResourceContainer.cpp: * rendering/svg/RenderSVGResourceContainer.h: * rendering/svg/RenderSVGResourceFilter.cpp: * rendering/svg/RenderSVGResourceFilter.h: * rendering/svg/RenderSVGResourceFilterPrimitive.cpp: * rendering/svg/RenderSVGResourceFilterPrimitive.h: * rendering/svg/RenderSVGResourceGradient.cpp: * rendering/svg/RenderSVGResourceGradient.h: * rendering/svg/RenderSVGResourceLinearGradient.cpp: * rendering/svg/RenderSVGResourceLinearGradient.h: * rendering/svg/RenderSVGResourceMarker.cpp: * rendering/svg/RenderSVGResourceMarker.h: * rendering/svg/RenderSVGResourceMasker.cpp: * rendering/svg/RenderSVGResourceMasker.h: * rendering/svg/RenderSVGResourcePattern.cpp: * rendering/svg/RenderSVGResourcePattern.h: * rendering/svg/RenderSVGResourceRadialGradient.cpp: * rendering/svg/RenderSVGResourceRadialGradient.h: * rendering/svg/RenderSVGRoot.cpp: * rendering/svg/RenderSVGRoot.h: * rendering/svg/RenderSVGShape.cpp: * rendering/svg/RenderSVGShape.h: * rendering/svg/RenderSVGTSpan.cpp: Added. * rendering/svg/RenderSVGTSpan.h: * rendering/svg/RenderSVGText.cpp: * rendering/svg/RenderSVGText.h: * rendering/svg/RenderSVGTextPath.cpp: * rendering/svg/RenderSVGTextPath.h: * rendering/svg/RenderSVGTransformableContainer.cpp: * rendering/svg/RenderSVGTransformableContainer.h: * rendering/svg/RenderSVGViewportContainer.cpp: * rendering/svg/RenderSVGViewportContainer.h: Source/WTF: This just adds an easy way of using the bmalloc IsoHeap API in WebKit. If bmalloc is not enabled, these macros will just make the object FastMalloced. * WTF.xcodeproj/project.pbxproj: * wtf/FastTLS.h: * wtf/IsoMalloc.h: Added. * wtf/IsoMallocInlines.h: Added. Canonical link: https://commits.webkit.org/195445@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@224537 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-11-07 19:21:52 +00:00
/*
* Copyright (C) 2017 Apple Inc. All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
* EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
* PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR
* CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
* EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
* PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
* OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#pragma once
#include "FixedVector.h"
#include "IsoPage.h"
bmalloc StaticMutex's constructor should be constexpr https://bugs.webkit.org/show_bug.cgi?id=180600 Reviewed by Mark Lam. StaticMutex and Mutex can be unified. This patch changes std::atomic_flag in StaticMutex to std::atomic<bool> to add constexpr constructor to StaticMutex. Then, StaticMutex can be initialized in static storage without calling any static initializers. And we also rename StaticMutex to Mutex simply. * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllIsoHeaps.cpp: (bmalloc::AllIsoHeaps::AllIsoHeaps): * bmalloc/AllIsoHeaps.h: * bmalloc/Allocator.cpp: (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): (bmalloc::Allocator::reallocate): (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::allocateLarge): * bmalloc/CryptoRandom.cpp: (bmalloc::ARC4RandomNumberGenerator::ARC4RandomNumberGenerator): * bmalloc/Deallocator.cpp: (bmalloc::Deallocator::scavenge): (bmalloc::Deallocator::processObjectLog): (bmalloc::Deallocator::deallocateSlowCase): * bmalloc/Deallocator.h: (bmalloc::Deallocator::lineCache): * bmalloc/DebugHeap.cpp: (bmalloc::DebugHeap::DebugHeap): * bmalloc/DebugHeap.h: * bmalloc/Environment.cpp: (bmalloc::Environment::Environment): * bmalloc/Environment.h: * bmalloc/Gigacage.cpp: (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::freeableMemory): (bmalloc::Heap::scavenge): (bmalloc::Heap::deallocateLineCache): (bmalloc::Heap::allocateSmallChunk): (bmalloc::Heap::allocateSmallPage): (bmalloc::Heap::deallocateSmallLine): (bmalloc::Heap::allocateSmallBumpRangesByMetadata): (bmalloc::Heap::allocateSmallBumpRangesByObject): (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::isLarge): (bmalloc::Heap::largeSize): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): (bmalloc::Heap::externalCommit): (bmalloc::Heap::externalDecommit): * bmalloc/Heap.h: (bmalloc::Heap::mutex): (bmalloc::Heap::allocateSmallBumpRanges): (bmalloc::Heap::derefSmallLine): * bmalloc/IsoDeallocator.h: * bmalloc/IsoHeap.h: * bmalloc/IsoTLSDeallocatorEntry.h: * bmalloc/IsoTLSDeallocatorEntryInlines.h: (bmalloc::IsoTLSDeallocatorEntry<Config>::IsoTLSDeallocatorEntry): * bmalloc/IsoTLSInlines.h: (bmalloc::IsoTLS::ensureHeap): * bmalloc/IsoTLSLayout.cpp: (bmalloc::IsoTLSLayout::IsoTLSLayout): (bmalloc::IsoTLSLayout::add): * bmalloc/IsoTLSLayout.h: * bmalloc/Mutex.cpp: Renamed from Source/bmalloc/bmalloc/StaticMutex.cpp. (bmalloc::Mutex::lockSlowCase): * bmalloc/Mutex.h: (bmalloc::sleep): (bmalloc::waitUntilFalse): (bmalloc::Mutex::try_lock): (bmalloc::Mutex::lock): (bmalloc::Mutex::unlock): (bmalloc::Mutex::Mutex): Deleted. * bmalloc/ObjectType.cpp: (bmalloc::objectType): * bmalloc/PerProcess.cpp: (bmalloc::getPerProcessData): * bmalloc/PerProcess.h: (bmalloc::PerProcess::mutex): (bmalloc::PerProcess::getSlowCase): * bmalloc/Scavenger.cpp: (bmalloc::Scavenger::Scavenger): (bmalloc::Scavenger::scavenge): (bmalloc::Scavenger::freeableMemory): * bmalloc/Scavenger.h: * bmalloc/SmallLine.h: (bmalloc::SmallLine::refCount): (bmalloc::SmallLine::ref): (bmalloc::SmallLine::deref): * bmalloc/SmallPage.h: (bmalloc::SmallPage::refCount): (bmalloc::SmallPage::hasFreeLines const): (bmalloc::SmallPage::setHasFreeLines): (bmalloc::SmallPage::ref): (bmalloc::SmallPage::deref): * bmalloc/StaticMutex.h: Removed. * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::VMHeap): * bmalloc/VMHeap.h: * bmalloc/Zone.cpp: (bmalloc::Zone::Zone): * bmalloc/Zone.h: * bmalloc/bmalloc.cpp: (bmalloc::api::tryLargeZeroedMemalignVirtual): (bmalloc::api::freeLargeVirtual): (bmalloc::api::isEnabled): (bmalloc::api::setScavengerThreadQOSClass): * bmalloc/bmalloc.h: Canonical link: https://commits.webkit.org/199892@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@230308 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2018-04-05 18:07:28 +00:00
#include "Mutex.h"
bmalloc should support strictly type-segregated isolated heaps https://bugs.webkit.org/show_bug.cgi?id=178108 Reviewed by Saam Barati, Simon Fraser, and Ryosuke Niwa. Source/bmalloc: This introduces a new allocation API in bmalloc called IsoHeap. An IsoHeap is templatized by type and created in static storage. When unused, it takes only a few words. When you do use it, each IsoHeap gets a bag of virtual pages unique to it. This prevents use-after-free bugs in one IsoHeap from affecting any other memory. At worst, two pointers of the same type will point to the same object even though they should not have. IsoHeaps allocate using a first-fit discipline that combines ideas from bmalloc and Riptide (the JSC GC): Like Riptide, it uses a bump'n'pop allocator. What Riptide calls blocks, IsoHeaps calls pages. Pages are collected into directories. Directories track pages using bitvectors, so that it's easy to quickly find a completely free page or one that has at least one free object. I think that the bump'n'pop allocator is as fast as the bmalloc Immix-style (page and line) allocator, but is better at allocating in holes. It's guaranteed to follow a first-fit discipline. However, the real reason why I wrote it that was is that this is what I'm more familiar with. This is a part of the design I want to revisit (bug 179278). Like bmalloc, it uses a deallocation log. This means that the internal IsoHeap data structures can be locked with a coarse-grained lock, since the deallocator only grabs it when flushing the log. Similarly, the allocator only grabs it when refilling the bump'n'pop FreeList. This adds a unit test for IsoHeaps. In this change, IsoHeaps are adopted only by WebCore's RenderObject. Note that despite the use of GC concepts, it's not a goal to make this code directly sharable with GC. The GC will probably have to do isolated heaps its own way (likely a special Subspace or something like that). * bmalloc.xcodeproj/project.pbxproj: * bmalloc/Algorithm.h: (bmalloc::findBitInWord): * bmalloc/AllIsoHeaps.cpp: Added. (bmalloc::AllIsoHeaps::AllIsoHeaps): (bmalloc::AllIsoHeaps::add): (bmalloc::AllIsoHeaps::head): * bmalloc/AllIsoHeaps.h: Added. * bmalloc/AllIsoHeapsInlines.h: Added. (bmalloc::AllIsoHeaps::forEach): * bmalloc/BMalloced.h: Added. * bmalloc/Bits.h: Added. (bmalloc::bitsArrayLength): (bmalloc::BitsWordView::BitsWordView): (bmalloc::BitsWordView::numBits const): (bmalloc::BitsWordView::word const): (bmalloc::BitsWordOwner::BitsWordOwner): (bmalloc::BitsWordOwner::view const): (bmalloc::BitsWordOwner::operator=): (bmalloc::BitsWordOwner::setAll): (bmalloc::BitsWordOwner::clearAll): (bmalloc::BitsWordOwner::set): (bmalloc::BitsWordOwner::numBits const): (bmalloc::BitsWordOwner::arrayLength const): (bmalloc::BitsWordOwner::word const): (bmalloc::BitsWordOwner::word): (bmalloc::BitsWordOwner::words const): (bmalloc::BitsWordOwner::words): (bmalloc::BitsAndWords::BitsAndWords): (bmalloc::BitsAndWords::view const): (bmalloc::BitsAndWords::numBits const): (bmalloc::BitsAndWords::word const): (bmalloc::BitsOrWords::BitsOrWords): (bmalloc::BitsOrWords::view const): (bmalloc::BitsOrWords::numBits const): (bmalloc::BitsOrWords::word const): (bmalloc::BitsNotWords::BitsNotWords): (bmalloc::BitsNotWords::view const): (bmalloc::BitsNotWords::numBits const): (bmalloc::BitsNotWords::word const): (bmalloc::BitsImpl::BitsImpl): (bmalloc::BitsImpl::numBits const): (bmalloc::BitsImpl::size const): (bmalloc::BitsImpl::arrayLength const): (bmalloc::BitsImpl::operator== const): (bmalloc::BitsImpl::operator!= const): (bmalloc::BitsImpl::at const): (bmalloc::BitsImpl::operator[] const): (bmalloc::BitsImpl::isEmpty const): (bmalloc::BitsImpl::operator& const): (bmalloc::BitsImpl::operator| const): (bmalloc::BitsImpl::operator~ const): (bmalloc::BitsImpl::forEachSetBit const): (bmalloc::BitsImpl::forEachClearBit const): (bmalloc::BitsImpl::forEachBit const): (bmalloc::BitsImpl::findBit const): (bmalloc::BitsImpl::findSetBit const): (bmalloc::BitsImpl::findClearBit const): (bmalloc::BitsImpl::wordView const): (bmalloc::BitsImpl::atImpl const): (bmalloc::Bits::Bits): (bmalloc::Bits::operator=): (bmalloc::Bits::resize): (bmalloc::Bits::setAll): (bmalloc::Bits::clearAll): (bmalloc::Bits::setAndCheck): (bmalloc::Bits::operator|=): (bmalloc::Bits::operator&=): (bmalloc::Bits::at const): (bmalloc::Bits::operator[] const): (bmalloc::Bits::BitReference::BitReference): (bmalloc::Bits::BitReference::operator bool const): (bmalloc::Bits::BitReference::operator=): (bmalloc::Bits::at): (bmalloc::Bits::operator[]): * bmalloc/CryptoRandom.cpp: Replaced with Source/bmalloc/bmalloc/CryptoRandom.cpp. (bmalloc::cryptoRandom): * bmalloc/CryptoRandom.h: Replaced with Source/bmalloc/bmalloc/CryptoRandom.h. * bmalloc/DeferredDecommit.h: Added. * bmalloc/DeferredDecommitInlines.h: Added. (bmalloc::DeferredDecommit::DeferredDecommit): * bmalloc/DeferredTrigger.h: Added. (bmalloc::DeferredTrigger::DeferredTrigger): * bmalloc/DeferredTriggerInlines.h: Added. (bmalloc::DeferredTrigger<trigger>::didBecome): (bmalloc::DeferredTrigger<trigger>::handleDeferral): * bmalloc/EligibilityResult.h: Added. (bmalloc::EligibilityResult::EligibilityResult): * bmalloc/EligibilityResultInlines.h: Added. (bmalloc::EligibilityResult<Config>::EligibilityResult): * bmalloc/FixedVector.h: * bmalloc/FreeList.cpp: Added. (bmalloc::FreeList::FreeList): (bmalloc::FreeList::~FreeList): (bmalloc::FreeList::clear): (bmalloc::FreeList::initializeList): (bmalloc::FreeList::initializeBump): (bmalloc::FreeList::contains const): * bmalloc/FreeList.h: Added. (bmalloc::FreeCell::scramble): (bmalloc::FreeCell::descramble): (bmalloc::FreeCell::setNext): (bmalloc::FreeCell::next const): (bmalloc::FreeList::allocationWillFail const): (bmalloc::FreeList::allocationWillSucceed const): (bmalloc::FreeList::originalSize const): (bmalloc::FreeList::head const): * bmalloc/FreeListInlines.h: Added. (bmalloc::FreeList::allocate): (bmalloc::FreeList::forEach const): * bmalloc/IsoAllocator.h: Added. * bmalloc/IsoAllocatorInlines.h: Added. (bmalloc::IsoAllocator<Config>::IsoAllocator): (bmalloc::IsoAllocator<Config>::~IsoAllocator): (bmalloc::IsoAllocator<Config>::allocate): (bmalloc::IsoAllocator<Config>::allocateSlow): (bmalloc::IsoAllocator<Config>::scavenge): * bmalloc/IsoConfig.h: Added. * bmalloc/IsoDeallocator.h: Added. * bmalloc/IsoDeallocatorInlines.h: Added. (bmalloc::IsoDeallocator<Config>::IsoDeallocator): (bmalloc::IsoDeallocator<Config>::~IsoDeallocator): (bmalloc::IsoDeallocator<Config>::deallocate): (bmalloc::IsoDeallocator<Config>::scavenge): * bmalloc/IsoDirectory.h: Added. (bmalloc::IsoDirectoryBaseBase::IsoDirectoryBaseBase): (bmalloc::IsoDirectoryBaseBase::~IsoDirectoryBaseBase): (bmalloc::IsoDirectoryBase::heap): * bmalloc/IsoDirectoryInlines.h: Added. (bmalloc::IsoDirectoryBase<Config>::IsoDirectoryBase): (bmalloc::passedNumPages>::IsoDirectory): (bmalloc::passedNumPages>::takeFirstEligible): (bmalloc::passedNumPages>::didBecome): (bmalloc::passedNumPages>::didDecommit): (bmalloc::passedNumPages>::scavenge): (bmalloc::passedNumPages>::forEachCommittedPage): * bmalloc/IsoDirectoryPage.h: Added. (bmalloc::IsoDirectoryPage::index const): * bmalloc/IsoDirectoryPageInlines.h: Added. (bmalloc::IsoDirectoryPage<Config>::IsoDirectoryPage): (bmalloc::IsoDirectoryPage<Config>::pageFor): * bmalloc/IsoHeap.h: Added. (bmalloc::api::IsoHeap::allocatorOffset): (bmalloc::api::IsoHeap::setAllocatorOffset): (bmalloc::api::IsoHeap::deallocatorOffset): (bmalloc::api::IsoHeap::setDeallocatorOffset): * bmalloc/IsoHeapImpl.cpp: Added. (bmalloc::IsoHeapImplBase::IsoHeapImplBase): (bmalloc::IsoHeapImplBase::~IsoHeapImplBase): (bmalloc::IsoHeapImplBase::scavengeNow): (bmalloc::IsoHeapImplBase::finishScavenging): * bmalloc/IsoHeapImpl.h: Added. * bmalloc/IsoHeapImplInlines.h: Added. (bmalloc::IsoHeapImpl<Config>::IsoHeapImpl): (bmalloc::IsoHeapImpl<Config>::takeFirstEligible): (bmalloc::IsoHeapImpl<Config>::didBecomeEligible): (bmalloc::IsoHeapImpl<Config>::scavenge): (bmalloc::IsoHeapImpl<Config>::allocatorOffset): (bmalloc::IsoHeapImpl<Config>::deallocatorOffset): (bmalloc::IsoHeapImpl<Config>::numLiveObjects): (bmalloc::IsoHeapImpl<Config>::numCommittedPages): (bmalloc::IsoHeapImpl<Config>::forEachDirectory): (bmalloc::IsoHeapImpl<Config>::forEachCommittedPage): (bmalloc::IsoHeapImpl<Config>::forEachLiveObject): * bmalloc/IsoHeapInlines.h: Added. (bmalloc::api::IsoHeap<Type>::allocate): (bmalloc::api::IsoHeap<Type>::tryAllocate): (bmalloc::api::IsoHeap<Type>::deallocate): (bmalloc::api::IsoHeap<Type>::scavenge): (bmalloc::api::IsoHeap<Type>::isInitialized): (bmalloc::api::IsoHeap<Type>::impl): * bmalloc/IsoPage.h: Added. (bmalloc::IsoPage::index const): (bmalloc::IsoPage::directory): (bmalloc::IsoPage::isInUseForAllocation const): (bmalloc::IsoPage::indexOfFirstObject): * bmalloc/IsoPageInlines.h: Added. (bmalloc::IsoPage<Config>::tryCreate): (bmalloc::IsoPage<Config>::IsoPage): (bmalloc::IsoPage<Config>::free): (bmalloc::IsoPage<Config>::startAllocating): (bmalloc::IsoPage<Config>::stopAllocating): (bmalloc::IsoPage<Config>::forEachLiveObject): * bmalloc/IsoPageTrigger.h: Added. * bmalloc/IsoTLS.cpp: Added. (bmalloc::IsoTLS::scavenge): (bmalloc::IsoTLS::IsoTLS): (bmalloc::IsoTLS::ensureEntries): (bmalloc::IsoTLS::destructor): (bmalloc::IsoTLS::sizeForCapacity): (bmalloc::IsoTLS::capacityForSize): (bmalloc::IsoTLS::size): (bmalloc::IsoTLS::forEachEntry): * bmalloc/IsoTLS.h: Added. * bmalloc/IsoTLSAllocatorEntry.h: Added. * bmalloc/IsoTLSAllocatorEntryInlines.h: Added. (bmalloc::IsoTLSAllocatorEntry<Config>::IsoTLSAllocatorEntry): (bmalloc::IsoTLSAllocatorEntry<Config>::~IsoTLSAllocatorEntry): (bmalloc::IsoTLSAllocatorEntry<Config>::construct): * bmalloc/IsoTLSDeallocatorEntry.h: Added. * bmalloc/IsoTLSDeallocatorEntryInlines.h: Added. (bmalloc::IsoTLSDeallocatorEntry<Config>::IsoTLSDeallocatorEntry): (bmalloc::IsoTLSDeallocatorEntry<Config>::~IsoTLSDeallocatorEntry): (bmalloc::IsoTLSDeallocatorEntry<Config>::construct): * bmalloc/IsoTLSEntry.cpp: Added. (bmalloc::IsoTLSEntry::IsoTLSEntry): (bmalloc::IsoTLSEntry::~IsoTLSEntry): * bmalloc/IsoTLSEntry.h: Added. (bmalloc::IsoTLSEntry::offset const): (bmalloc::IsoTLSEntry::alignment const): (bmalloc::IsoTLSEntry::size const): (bmalloc::IsoTLSEntry::extent const): * bmalloc/IsoTLSEntryInlines.h: Added. (bmalloc::IsoTLSEntry::walkUpToInclusive): (bmalloc::DefaultIsoTLSEntry<EntryType>::DefaultIsoTLSEntry): (bmalloc::DefaultIsoTLSEntry<EntryType>::~DefaultIsoTLSEntry): (bmalloc::DefaultIsoTLSEntry<EntryType>::move): (bmalloc::DefaultIsoTLSEntry<EntryType>::destruct): (bmalloc::DefaultIsoTLSEntry<EntryType>::scavenge): * bmalloc/IsoTLSInlines.h: Added. (bmalloc::IsoTLS::allocate): (bmalloc::IsoTLS::deallocate): (bmalloc::IsoTLS::scavenge): (bmalloc::IsoTLS::allocator): (bmalloc::IsoTLS::deallocator): (bmalloc::IsoTLS::get): (bmalloc::IsoTLS::set): (bmalloc::IsoTLS::ensureHeap): (bmalloc::IsoTLS::ensureHeapAndEntries): * bmalloc/IsoTLSLayout.cpp: Added. (bmalloc::IsoTLSLayout::IsoTLSLayout): (bmalloc::IsoTLSLayout::add): * bmalloc/IsoTLSLayout.h: Added. (bmalloc::IsoTLSLayout::head const): * bmalloc/PerHeapKind.h: * bmalloc/PerProcess.h: (bmalloc::PerProcess<T>::getFastCase): * bmalloc/Scavenger.cpp: (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: * bmalloc/bmalloc.h: (bmalloc::api::scavengeThisThread): * test: Added. * test/testbmalloc.cpp: Added. (hiddenTruthBecauseNoReturnIsStupid): (usage): (assertEmptyPointerSet): (assertHasObjects): (assertHasOnlyObjects): (assertClean): (testIsoSimple): (testIsoSimpleScavengeBeforeDealloc): (testIsoFlipFlopFragmentedPages): (testIsoFlipFlopFragmentedPagesScavengeInMiddle): (BisoMalloced::BisoMalloced): (testBisoMalloced): (BisoMallocedInline::BisoMallocedInline): (testBisoMallocedInline): (run): (main): Source/WebCore: No new tests because no new change in behavior. Though, the bmalloc change has a unit test. Adopting IsoHeap means dropping in macros in both the .h and .cpp file of each class that we opt in. It's not pretty, but it helps ensure speedy allocation since it means that we never have to do any kind of switch or dynamic lookup to find the right allocator for a type. This change is perf-neutral on MotionMark, PLT3, and membuster. * Sources.txt: * html/shadow/SliderThumbElement.cpp: * html/shadow/SliderThumbElement.h: * html/shadow/mac/ImageControlsButtonElementMac.cpp: * html/shadow/mac/ImageControlsRootElementMac.cpp: * rendering/RenderAttachment.cpp: * rendering/RenderAttachment.h: * rendering/RenderBlock.cpp: * rendering/RenderBlock.h: * rendering/RenderBlockFlow.cpp: * rendering/RenderBlockFlow.h: * rendering/RenderBox.cpp: * rendering/RenderBox.h: * rendering/RenderBoxModelObject.cpp: * rendering/RenderBoxModelObject.h: * rendering/RenderButton.cpp: * rendering/RenderButton.h: * rendering/RenderCombineText.cpp: * rendering/RenderCombineText.h: * rendering/RenderCounter.cpp: * rendering/RenderCounter.h: * rendering/RenderDeprecatedFlexibleBox.cpp: * rendering/RenderDeprecatedFlexibleBox.h: * rendering/RenderDetailsMarker.cpp: * rendering/RenderDetailsMarker.h: * rendering/RenderElement.cpp: * rendering/RenderElement.h: * rendering/RenderEmbeddedObject.cpp: * rendering/RenderEmbeddedObject.h: * rendering/RenderFileUploadControl.cpp: * rendering/RenderFileUploadControl.h: * rendering/RenderFlexibleBox.cpp: * rendering/RenderFlexibleBox.h: * rendering/RenderFragmentContainer.cpp: * rendering/RenderFragmentContainer.h: * rendering/RenderFragmentContainerSet.cpp: * rendering/RenderFragmentContainerSet.h: * rendering/RenderFragmentedFlow.cpp: * rendering/RenderFragmentedFlow.h: * rendering/RenderFrameBase.cpp: * rendering/RenderFrameBase.h: * rendering/RenderFrameSet.cpp: * rendering/RenderFrameSet.h: * rendering/RenderFullScreen.cpp: * rendering/RenderFullScreen.h: * rendering/RenderGrid.cpp: * rendering/RenderGrid.h: * rendering/RenderHTMLCanvas.cpp: * rendering/RenderHTMLCanvas.h: * rendering/RenderImage.cpp: * rendering/RenderImage.h: * rendering/RenderImageResourceStyleImage.cpp: * rendering/RenderImageResourceStyleImage.h: * rendering/RenderInline.cpp: * rendering/RenderInline.h: * rendering/RenderLayerModelObject.cpp: * rendering/RenderLayerModelObject.h: * rendering/RenderLineBreak.cpp: * rendering/RenderLineBreak.h: * rendering/RenderListBox.cpp: * rendering/RenderListBox.h: * rendering/RenderListItem.cpp: * rendering/RenderListItem.h: * rendering/RenderListMarker.cpp: * rendering/RenderListMarker.h: * rendering/RenderMedia.cpp: * rendering/RenderMedia.h: * rendering/RenderMediaControlElements.cpp: * rendering/RenderMediaControlElements.h: * rendering/RenderMenuList.cpp: * rendering/RenderMenuList.h: * rendering/RenderMeter.cpp: * rendering/RenderMeter.h: * rendering/RenderMultiColumnFlow.cpp: * rendering/RenderMultiColumnFlow.h: * rendering/RenderMultiColumnSet.cpp: * rendering/RenderMultiColumnSet.h: * rendering/RenderMultiColumnSpannerPlaceholder.cpp: * rendering/RenderMultiColumnSpannerPlaceholder.h: * rendering/RenderObject.cpp: * rendering/RenderObject.h: * rendering/RenderProgress.cpp: * rendering/RenderProgress.h: * rendering/RenderQuote.cpp: * rendering/RenderQuote.h: * rendering/RenderReplaced.cpp: * rendering/RenderReplaced.h: * rendering/RenderReplica.cpp: * rendering/RenderReplica.h: * rendering/RenderRuby.cpp: * rendering/RenderRuby.h: * rendering/RenderRubyBase.cpp: * rendering/RenderRubyBase.h: * rendering/RenderRubyRun.cpp: * rendering/RenderRubyRun.h: * rendering/RenderRubyText.cpp: * rendering/RenderRubyText.h: * rendering/RenderScrollbarPart.cpp: * rendering/RenderScrollbarPart.h: * rendering/RenderSearchField.cpp: * rendering/RenderSearchField.h: * rendering/RenderSlider.cpp: * rendering/RenderSlider.h: * rendering/RenderTable.cpp: * rendering/RenderTable.h: * rendering/RenderTableCaption.cpp: * rendering/RenderTableCaption.h: * rendering/RenderTableCell.cpp: * rendering/RenderTableCell.h: * rendering/RenderTableCol.cpp: * rendering/RenderTableCol.h: * rendering/RenderTableRow.cpp: * rendering/RenderTableRow.h: * rendering/RenderTableSection.cpp: * rendering/RenderTableSection.h: * rendering/RenderText.cpp: * rendering/RenderText.h: * rendering/RenderTextControl.cpp: * rendering/RenderTextControl.h: * rendering/RenderTextControlMultiLine.cpp: * rendering/RenderTextControlMultiLine.h: * rendering/RenderTextControlSingleLine.cpp: * rendering/RenderTextControlSingleLine.h: * rendering/RenderTextFragment.cpp: * rendering/RenderTextFragment.h: * rendering/RenderVTTCue.cpp: * rendering/RenderVTTCue.h: * rendering/RenderVideo.cpp: * rendering/RenderVideo.h: * rendering/RenderView.cpp: * rendering/RenderView.h: * rendering/RenderWidget.cpp: * rendering/RenderWidget.h: * rendering/mathml/RenderMathMLBlock.cpp: * rendering/mathml/RenderMathMLBlock.h: * rendering/mathml/RenderMathMLFenced.cpp: * rendering/mathml/RenderMathMLFenced.h: * rendering/mathml/RenderMathMLFencedOperator.cpp: * rendering/mathml/RenderMathMLFencedOperator.h: * rendering/mathml/RenderMathMLFraction.cpp: * rendering/mathml/RenderMathMLFraction.h: * rendering/mathml/RenderMathMLMath.cpp: * rendering/mathml/RenderMathMLMath.h: * rendering/mathml/RenderMathMLMenclose.cpp: * rendering/mathml/RenderMathMLMenclose.h: * rendering/mathml/RenderMathMLOperator.cpp: * rendering/mathml/RenderMathMLOperator.h: * rendering/mathml/RenderMathMLPadded.cpp: * rendering/mathml/RenderMathMLPadded.h: * rendering/mathml/RenderMathMLRoot.cpp: * rendering/mathml/RenderMathMLRoot.h: * rendering/mathml/RenderMathMLRow.cpp: * rendering/mathml/RenderMathMLRow.h: * rendering/mathml/RenderMathMLScripts.cpp: * rendering/mathml/RenderMathMLScripts.h: * rendering/mathml/RenderMathMLSpace.cpp: * rendering/mathml/RenderMathMLSpace.h: * rendering/mathml/RenderMathMLToken.cpp: * rendering/mathml/RenderMathMLToken.h: * rendering/mathml/RenderMathMLUnderOver.cpp: * rendering/mathml/RenderMathMLUnderOver.h: * rendering/svg/RenderSVGBlock.cpp: * rendering/svg/RenderSVGBlock.h: * rendering/svg/RenderSVGContainer.cpp: * rendering/svg/RenderSVGContainer.h: * rendering/svg/RenderSVGEllipse.cpp: * rendering/svg/RenderSVGEllipse.h: * rendering/svg/RenderSVGForeignObject.cpp: * rendering/svg/RenderSVGForeignObject.h: * rendering/svg/RenderSVGGradientStop.cpp: * rendering/svg/RenderSVGGradientStop.h: * rendering/svg/RenderSVGHiddenContainer.cpp: * rendering/svg/RenderSVGHiddenContainer.h: * rendering/svg/RenderSVGImage.cpp: * rendering/svg/RenderSVGImage.h: * rendering/svg/RenderSVGInline.cpp: * rendering/svg/RenderSVGInline.h: * rendering/svg/RenderSVGInlineText.cpp: * rendering/svg/RenderSVGInlineText.h: * rendering/svg/RenderSVGModelObject.cpp: * rendering/svg/RenderSVGModelObject.h: * rendering/svg/RenderSVGPath.cpp: * rendering/svg/RenderSVGPath.h: * rendering/svg/RenderSVGRect.cpp: * rendering/svg/RenderSVGRect.h: * rendering/svg/RenderSVGResourceClipper.cpp: * rendering/svg/RenderSVGResourceClipper.h: * rendering/svg/RenderSVGResourceContainer.cpp: * rendering/svg/RenderSVGResourceContainer.h: * rendering/svg/RenderSVGResourceFilter.cpp: * rendering/svg/RenderSVGResourceFilter.h: * rendering/svg/RenderSVGResourceFilterPrimitive.cpp: * rendering/svg/RenderSVGResourceFilterPrimitive.h: * rendering/svg/RenderSVGResourceGradient.cpp: * rendering/svg/RenderSVGResourceGradient.h: * rendering/svg/RenderSVGResourceLinearGradient.cpp: * rendering/svg/RenderSVGResourceLinearGradient.h: * rendering/svg/RenderSVGResourceMarker.cpp: * rendering/svg/RenderSVGResourceMarker.h: * rendering/svg/RenderSVGResourceMasker.cpp: * rendering/svg/RenderSVGResourceMasker.h: * rendering/svg/RenderSVGResourcePattern.cpp: * rendering/svg/RenderSVGResourcePattern.h: * rendering/svg/RenderSVGResourceRadialGradient.cpp: * rendering/svg/RenderSVGResourceRadialGradient.h: * rendering/svg/RenderSVGRoot.cpp: * rendering/svg/RenderSVGRoot.h: * rendering/svg/RenderSVGShape.cpp: * rendering/svg/RenderSVGShape.h: * rendering/svg/RenderSVGTSpan.cpp: Added. * rendering/svg/RenderSVGTSpan.h: * rendering/svg/RenderSVGText.cpp: * rendering/svg/RenderSVGText.h: * rendering/svg/RenderSVGTextPath.cpp: * rendering/svg/RenderSVGTextPath.h: * rendering/svg/RenderSVGTransformableContainer.cpp: * rendering/svg/RenderSVGTransformableContainer.h: * rendering/svg/RenderSVGViewportContainer.cpp: * rendering/svg/RenderSVGViewportContainer.h: Source/WTF: This just adds an easy way of using the bmalloc IsoHeap API in WebKit. If bmalloc is not enabled, these macros will just make the object FastMalloced. * WTF.xcodeproj/project.pbxproj: * wtf/FastTLS.h: * wtf/IsoMalloc.h: Added. * wtf/IsoMallocInlines.h: Added. Canonical link: https://commits.webkit.org/195445@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@224537 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-11-07 19:21:52 +00:00
namespace bmalloc {
template<typename Config>
class IsoDeallocator {
public:
static constexpr unsigned logsPerPage = 2;
static constexpr unsigned logCapacity =
(IsoPageBase::pageSize / Config::objectSize + logsPerPage - 1) / logsPerPage;
IsoDeallocator(Mutex& lock);
~IsoDeallocator();
[bmalloc] IsoHeap should have lower tier using shared IsoPage https://bugs.webkit.org/show_bug.cgi?id=196837 Reviewed by Filip Pizlo. IsoHeap had a scalability problem. Once one instance is allocated from IsoHeap, it immediately allocates 16KB page for this type. But some types allocate only a few instances. It leads to memory wastage, and it also limits the scalability of IsoHeap since we need to carefully select classes which will be confined in IsoHeap due to this characteristics. If we can remove this wastage, we can apply IsoHeap more aggressively without causing memory regression, this is the goal of this patch. In this patch, we introduce a slow tier to IsoHeap allocation. Initially, the allocator for a certain type allocates instances from a shared page with the other allocators, and eventually, the allocator tiers up and gets dedicated pages if instances of the type are allocated a lot. This "shared" tier is slow, but it is totally OK because we will tier up to the normal fast tier if allocation frequently happens. Even the instance is allocated from pages shared with the other allocators, we still make the allocated memory region dedicated to the specific type: once a memory region is allocated for a certain type from a shared page, this region continues being used only for this type even after this memory is freed. To summarize the changes: 1. We introduce "shared" tier to IsoHeap allocation. Up to N (N = 8 for now, but we can pick any power-of-two numbers up to 32) allocations, we continue using this tier. We allocate memory from shared pages so that we do not waste 16KB pages for types which only allocates a few instances. 2. We eventually tier up to the "fast" tier, and eventually tier down to the "shared" tier too. We measure the period between slow paths, and switch the appropriate tier for the type. Currently, we use 1 seconds as heuristics. We also count # of allocations per cycle to avoid pathological slow downs. 3. Shared page mechanism must keep the characteristics of IsoHeap. Once a memory region is allocated for a certain type, this memory region must be dedicated to this type. We keep track the allocated memory regions from shared pages in IsoHeapImpl, and ensure that we never reuse a memory region for a different type. This patch improves PLUM2 by 1.4% (128.4MB v.s. 126.62MB), and early Speedometer2 results are performance-neutral. * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/Algorithm.h: (bmalloc::roundUpToMultipleOfImpl): (bmalloc::roundUpToMultipleOf): * bmalloc/BCompiler.h: * bmalloc/BExport.h: * bmalloc/FreeList.h: * bmalloc/IsoAllocator.h: * bmalloc/IsoAllocatorInlines.h: (bmalloc::IsoAllocator<Config>::allocateSlow): * bmalloc/IsoDeallocator.h: * bmalloc/IsoDeallocatorInlines.h: (bmalloc::IsoDeallocator<Config>::deallocate): * bmalloc/IsoHeapImpl.h: * bmalloc/IsoHeapImplInlines.h: (bmalloc::IsoHeapImpl<Config>::scavenge): (bmalloc::IsoHeapImpl<Config>::forEachLiveObject): (bmalloc::IsoHeapImpl<Config>::updateAllocationMode): (bmalloc::IsoHeapImpl<Config>::allocateFromShared): * bmalloc/IsoPage.h: (bmalloc::IsoPageBase::IsoPageBase): (bmalloc::IsoPageBase::isShared const): * bmalloc/IsoPageInlines.h: (bmalloc::IsoPage<Config>::IsoPage): (bmalloc::IsoPageBase::pageFor): (bmalloc::IsoPage<Config>::pageFor): (bmalloc::IsoPage<Config>::free): * bmalloc/IsoSharedConfig.h: Copied from Source/bmalloc/bmalloc/BExport.h. * bmalloc/IsoSharedHeap.cpp: Copied from Source/bmalloc/bmalloc/BExport.h. * bmalloc/IsoSharedHeap.h: Copied from Source/bmalloc/bmalloc/IsoAllocator.h. (bmalloc::VariadicBumpAllocator::VariadicBumpAllocator): (bmalloc::IsoSharedHeap::IsoSharedHeap): * bmalloc/IsoSharedHeapInlines.h: Added. (bmalloc::VariadicBumpAllocator::allocate): (bmalloc::IsoSharedHeap::allocateNew): (bmalloc::IsoSharedHeap::allocateSlow): * bmalloc/IsoSharedPage.cpp: Copied from Source/bmalloc/bmalloc/BExport.h. (bmalloc::IsoSharedPage::tryCreate): * bmalloc/IsoSharedPage.h: Copied from Source/bmalloc/bmalloc/IsoDeallocator.h. (bmalloc::IsoSharedPage::IsoSharedPage): (bmalloc::indexSlotFor): * bmalloc/IsoSharedPageInlines.h: Added. (bmalloc::IsoSharedPage::free): (bmalloc::IsoSharedPage::startAllocating): (bmalloc::IsoSharedPage::stopAllocating): * bmalloc/IsoTLS.h: * bmalloc/IsoTLSInlines.h: (bmalloc::IsoTLS::deallocateImpl): (bmalloc::IsoTLS::deallocateFast): (bmalloc::IsoTLS::deallocateSlow): * bmalloc/StdLibExtras.h: (bmalloc::bitwise_cast): * test/testbmalloc.cpp: (testIsoMallocAndFreeFast): (run): Canonical link: https://commits.webkit.org/211356@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@244481 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2019-04-20 03:29:40 +00:00
template<typename Type>
void deallocate(api::IsoHeap<Type>&, void* p);
bmalloc should support strictly type-segregated isolated heaps https://bugs.webkit.org/show_bug.cgi?id=178108 Reviewed by Saam Barati, Simon Fraser, and Ryosuke Niwa. Source/bmalloc: This introduces a new allocation API in bmalloc called IsoHeap. An IsoHeap is templatized by type and created in static storage. When unused, it takes only a few words. When you do use it, each IsoHeap gets a bag of virtual pages unique to it. This prevents use-after-free bugs in one IsoHeap from affecting any other memory. At worst, two pointers of the same type will point to the same object even though they should not have. IsoHeaps allocate using a first-fit discipline that combines ideas from bmalloc and Riptide (the JSC GC): Like Riptide, it uses a bump'n'pop allocator. What Riptide calls blocks, IsoHeaps calls pages. Pages are collected into directories. Directories track pages using bitvectors, so that it's easy to quickly find a completely free page or one that has at least one free object. I think that the bump'n'pop allocator is as fast as the bmalloc Immix-style (page and line) allocator, but is better at allocating in holes. It's guaranteed to follow a first-fit discipline. However, the real reason why I wrote it that was is that this is what I'm more familiar with. This is a part of the design I want to revisit (bug 179278). Like bmalloc, it uses a deallocation log. This means that the internal IsoHeap data structures can be locked with a coarse-grained lock, since the deallocator only grabs it when flushing the log. Similarly, the allocator only grabs it when refilling the bump'n'pop FreeList. This adds a unit test for IsoHeaps. In this change, IsoHeaps are adopted only by WebCore's RenderObject. Note that despite the use of GC concepts, it's not a goal to make this code directly sharable with GC. The GC will probably have to do isolated heaps its own way (likely a special Subspace or something like that). * bmalloc.xcodeproj/project.pbxproj: * bmalloc/Algorithm.h: (bmalloc::findBitInWord): * bmalloc/AllIsoHeaps.cpp: Added. (bmalloc::AllIsoHeaps::AllIsoHeaps): (bmalloc::AllIsoHeaps::add): (bmalloc::AllIsoHeaps::head): * bmalloc/AllIsoHeaps.h: Added. * bmalloc/AllIsoHeapsInlines.h: Added. (bmalloc::AllIsoHeaps::forEach): * bmalloc/BMalloced.h: Added. * bmalloc/Bits.h: Added. (bmalloc::bitsArrayLength): (bmalloc::BitsWordView::BitsWordView): (bmalloc::BitsWordView::numBits const): (bmalloc::BitsWordView::word const): (bmalloc::BitsWordOwner::BitsWordOwner): (bmalloc::BitsWordOwner::view const): (bmalloc::BitsWordOwner::operator=): (bmalloc::BitsWordOwner::setAll): (bmalloc::BitsWordOwner::clearAll): (bmalloc::BitsWordOwner::set): (bmalloc::BitsWordOwner::numBits const): (bmalloc::BitsWordOwner::arrayLength const): (bmalloc::BitsWordOwner::word const): (bmalloc::BitsWordOwner::word): (bmalloc::BitsWordOwner::words const): (bmalloc::BitsWordOwner::words): (bmalloc::BitsAndWords::BitsAndWords): (bmalloc::BitsAndWords::view const): (bmalloc::BitsAndWords::numBits const): (bmalloc::BitsAndWords::word const): (bmalloc::BitsOrWords::BitsOrWords): (bmalloc::BitsOrWords::view const): (bmalloc::BitsOrWords::numBits const): (bmalloc::BitsOrWords::word const): (bmalloc::BitsNotWords::BitsNotWords): (bmalloc::BitsNotWords::view const): (bmalloc::BitsNotWords::numBits const): (bmalloc::BitsNotWords::word const): (bmalloc::BitsImpl::BitsImpl): (bmalloc::BitsImpl::numBits const): (bmalloc::BitsImpl::size const): (bmalloc::BitsImpl::arrayLength const): (bmalloc::BitsImpl::operator== const): (bmalloc::BitsImpl::operator!= const): (bmalloc::BitsImpl::at const): (bmalloc::BitsImpl::operator[] const): (bmalloc::BitsImpl::isEmpty const): (bmalloc::BitsImpl::operator& const): (bmalloc::BitsImpl::operator| const): (bmalloc::BitsImpl::operator~ const): (bmalloc::BitsImpl::forEachSetBit const): (bmalloc::BitsImpl::forEachClearBit const): (bmalloc::BitsImpl::forEachBit const): (bmalloc::BitsImpl::findBit const): (bmalloc::BitsImpl::findSetBit const): (bmalloc::BitsImpl::findClearBit const): (bmalloc::BitsImpl::wordView const): (bmalloc::BitsImpl::atImpl const): (bmalloc::Bits::Bits): (bmalloc::Bits::operator=): (bmalloc::Bits::resize): (bmalloc::Bits::setAll): (bmalloc::Bits::clearAll): (bmalloc::Bits::setAndCheck): (bmalloc::Bits::operator|=): (bmalloc::Bits::operator&=): (bmalloc::Bits::at const): (bmalloc::Bits::operator[] const): (bmalloc::Bits::BitReference::BitReference): (bmalloc::Bits::BitReference::operator bool const): (bmalloc::Bits::BitReference::operator=): (bmalloc::Bits::at): (bmalloc::Bits::operator[]): * bmalloc/CryptoRandom.cpp: Replaced with Source/bmalloc/bmalloc/CryptoRandom.cpp. (bmalloc::cryptoRandom): * bmalloc/CryptoRandom.h: Replaced with Source/bmalloc/bmalloc/CryptoRandom.h. * bmalloc/DeferredDecommit.h: Added. * bmalloc/DeferredDecommitInlines.h: Added. (bmalloc::DeferredDecommit::DeferredDecommit): * bmalloc/DeferredTrigger.h: Added. (bmalloc::DeferredTrigger::DeferredTrigger): * bmalloc/DeferredTriggerInlines.h: Added. (bmalloc::DeferredTrigger<trigger>::didBecome): (bmalloc::DeferredTrigger<trigger>::handleDeferral): * bmalloc/EligibilityResult.h: Added. (bmalloc::EligibilityResult::EligibilityResult): * bmalloc/EligibilityResultInlines.h: Added. (bmalloc::EligibilityResult<Config>::EligibilityResult): * bmalloc/FixedVector.h: * bmalloc/FreeList.cpp: Added. (bmalloc::FreeList::FreeList): (bmalloc::FreeList::~FreeList): (bmalloc::FreeList::clear): (bmalloc::FreeList::initializeList): (bmalloc::FreeList::initializeBump): (bmalloc::FreeList::contains const): * bmalloc/FreeList.h: Added. (bmalloc::FreeCell::scramble): (bmalloc::FreeCell::descramble): (bmalloc::FreeCell::setNext): (bmalloc::FreeCell::next const): (bmalloc::FreeList::allocationWillFail const): (bmalloc::FreeList::allocationWillSucceed const): (bmalloc::FreeList::originalSize const): (bmalloc::FreeList::head const): * bmalloc/FreeListInlines.h: Added. (bmalloc::FreeList::allocate): (bmalloc::FreeList::forEach const): * bmalloc/IsoAllocator.h: Added. * bmalloc/IsoAllocatorInlines.h: Added. (bmalloc::IsoAllocator<Config>::IsoAllocator): (bmalloc::IsoAllocator<Config>::~IsoAllocator): (bmalloc::IsoAllocator<Config>::allocate): (bmalloc::IsoAllocator<Config>::allocateSlow): (bmalloc::IsoAllocator<Config>::scavenge): * bmalloc/IsoConfig.h: Added. * bmalloc/IsoDeallocator.h: Added. * bmalloc/IsoDeallocatorInlines.h: Added. (bmalloc::IsoDeallocator<Config>::IsoDeallocator): (bmalloc::IsoDeallocator<Config>::~IsoDeallocator): (bmalloc::IsoDeallocator<Config>::deallocate): (bmalloc::IsoDeallocator<Config>::scavenge): * bmalloc/IsoDirectory.h: Added. (bmalloc::IsoDirectoryBaseBase::IsoDirectoryBaseBase): (bmalloc::IsoDirectoryBaseBase::~IsoDirectoryBaseBase): (bmalloc::IsoDirectoryBase::heap): * bmalloc/IsoDirectoryInlines.h: Added. (bmalloc::IsoDirectoryBase<Config>::IsoDirectoryBase): (bmalloc::passedNumPages>::IsoDirectory): (bmalloc::passedNumPages>::takeFirstEligible): (bmalloc::passedNumPages>::didBecome): (bmalloc::passedNumPages>::didDecommit): (bmalloc::passedNumPages>::scavenge): (bmalloc::passedNumPages>::forEachCommittedPage): * bmalloc/IsoDirectoryPage.h: Added. (bmalloc::IsoDirectoryPage::index const): * bmalloc/IsoDirectoryPageInlines.h: Added. (bmalloc::IsoDirectoryPage<Config>::IsoDirectoryPage): (bmalloc::IsoDirectoryPage<Config>::pageFor): * bmalloc/IsoHeap.h: Added. (bmalloc::api::IsoHeap::allocatorOffset): (bmalloc::api::IsoHeap::setAllocatorOffset): (bmalloc::api::IsoHeap::deallocatorOffset): (bmalloc::api::IsoHeap::setDeallocatorOffset): * bmalloc/IsoHeapImpl.cpp: Added. (bmalloc::IsoHeapImplBase::IsoHeapImplBase): (bmalloc::IsoHeapImplBase::~IsoHeapImplBase): (bmalloc::IsoHeapImplBase::scavengeNow): (bmalloc::IsoHeapImplBase::finishScavenging): * bmalloc/IsoHeapImpl.h: Added. * bmalloc/IsoHeapImplInlines.h: Added. (bmalloc::IsoHeapImpl<Config>::IsoHeapImpl): (bmalloc::IsoHeapImpl<Config>::takeFirstEligible): (bmalloc::IsoHeapImpl<Config>::didBecomeEligible): (bmalloc::IsoHeapImpl<Config>::scavenge): (bmalloc::IsoHeapImpl<Config>::allocatorOffset): (bmalloc::IsoHeapImpl<Config>::deallocatorOffset): (bmalloc::IsoHeapImpl<Config>::numLiveObjects): (bmalloc::IsoHeapImpl<Config>::numCommittedPages): (bmalloc::IsoHeapImpl<Config>::forEachDirectory): (bmalloc::IsoHeapImpl<Config>::forEachCommittedPage): (bmalloc::IsoHeapImpl<Config>::forEachLiveObject): * bmalloc/IsoHeapInlines.h: Added. (bmalloc::api::IsoHeap<Type>::allocate): (bmalloc::api::IsoHeap<Type>::tryAllocate): (bmalloc::api::IsoHeap<Type>::deallocate): (bmalloc::api::IsoHeap<Type>::scavenge): (bmalloc::api::IsoHeap<Type>::isInitialized): (bmalloc::api::IsoHeap<Type>::impl): * bmalloc/IsoPage.h: Added. (bmalloc::IsoPage::index const): (bmalloc::IsoPage::directory): (bmalloc::IsoPage::isInUseForAllocation const): (bmalloc::IsoPage::indexOfFirstObject): * bmalloc/IsoPageInlines.h: Added. (bmalloc::IsoPage<Config>::tryCreate): (bmalloc::IsoPage<Config>::IsoPage): (bmalloc::IsoPage<Config>::free): (bmalloc::IsoPage<Config>::startAllocating): (bmalloc::IsoPage<Config>::stopAllocating): (bmalloc::IsoPage<Config>::forEachLiveObject): * bmalloc/IsoPageTrigger.h: Added. * bmalloc/IsoTLS.cpp: Added. (bmalloc::IsoTLS::scavenge): (bmalloc::IsoTLS::IsoTLS): (bmalloc::IsoTLS::ensureEntries): (bmalloc::IsoTLS::destructor): (bmalloc::IsoTLS::sizeForCapacity): (bmalloc::IsoTLS::capacityForSize): (bmalloc::IsoTLS::size): (bmalloc::IsoTLS::forEachEntry): * bmalloc/IsoTLS.h: Added. * bmalloc/IsoTLSAllocatorEntry.h: Added. * bmalloc/IsoTLSAllocatorEntryInlines.h: Added. (bmalloc::IsoTLSAllocatorEntry<Config>::IsoTLSAllocatorEntry): (bmalloc::IsoTLSAllocatorEntry<Config>::~IsoTLSAllocatorEntry): (bmalloc::IsoTLSAllocatorEntry<Config>::construct): * bmalloc/IsoTLSDeallocatorEntry.h: Added. * bmalloc/IsoTLSDeallocatorEntryInlines.h: Added. (bmalloc::IsoTLSDeallocatorEntry<Config>::IsoTLSDeallocatorEntry): (bmalloc::IsoTLSDeallocatorEntry<Config>::~IsoTLSDeallocatorEntry): (bmalloc::IsoTLSDeallocatorEntry<Config>::construct): * bmalloc/IsoTLSEntry.cpp: Added. (bmalloc::IsoTLSEntry::IsoTLSEntry): (bmalloc::IsoTLSEntry::~IsoTLSEntry): * bmalloc/IsoTLSEntry.h: Added. (bmalloc::IsoTLSEntry::offset const): (bmalloc::IsoTLSEntry::alignment const): (bmalloc::IsoTLSEntry::size const): (bmalloc::IsoTLSEntry::extent const): * bmalloc/IsoTLSEntryInlines.h: Added. (bmalloc::IsoTLSEntry::walkUpToInclusive): (bmalloc::DefaultIsoTLSEntry<EntryType>::DefaultIsoTLSEntry): (bmalloc::DefaultIsoTLSEntry<EntryType>::~DefaultIsoTLSEntry): (bmalloc::DefaultIsoTLSEntry<EntryType>::move): (bmalloc::DefaultIsoTLSEntry<EntryType>::destruct): (bmalloc::DefaultIsoTLSEntry<EntryType>::scavenge): * bmalloc/IsoTLSInlines.h: Added. (bmalloc::IsoTLS::allocate): (bmalloc::IsoTLS::deallocate): (bmalloc::IsoTLS::scavenge): (bmalloc::IsoTLS::allocator): (bmalloc::IsoTLS::deallocator): (bmalloc::IsoTLS::get): (bmalloc::IsoTLS::set): (bmalloc::IsoTLS::ensureHeap): (bmalloc::IsoTLS::ensureHeapAndEntries): * bmalloc/IsoTLSLayout.cpp: Added. (bmalloc::IsoTLSLayout::IsoTLSLayout): (bmalloc::IsoTLSLayout::add): * bmalloc/IsoTLSLayout.h: Added. (bmalloc::IsoTLSLayout::head const): * bmalloc/PerHeapKind.h: * bmalloc/PerProcess.h: (bmalloc::PerProcess<T>::getFastCase): * bmalloc/Scavenger.cpp: (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: * bmalloc/bmalloc.h: (bmalloc::api::scavengeThisThread): * test: Added. * test/testbmalloc.cpp: Added. (hiddenTruthBecauseNoReturnIsStupid): (usage): (assertEmptyPointerSet): (assertHasObjects): (assertHasOnlyObjects): (assertClean): (testIsoSimple): (testIsoSimpleScavengeBeforeDealloc): (testIsoFlipFlopFragmentedPages): (testIsoFlipFlopFragmentedPagesScavengeInMiddle): (BisoMalloced::BisoMalloced): (testBisoMalloced): (BisoMallocedInline::BisoMallocedInline): (testBisoMallocedInline): (run): (main): Source/WebCore: No new tests because no new change in behavior. Though, the bmalloc change has a unit test. Adopting IsoHeap means dropping in macros in both the .h and .cpp file of each class that we opt in. It's not pretty, but it helps ensure speedy allocation since it means that we never have to do any kind of switch or dynamic lookup to find the right allocator for a type. This change is perf-neutral on MotionMark, PLT3, and membuster. * Sources.txt: * html/shadow/SliderThumbElement.cpp: * html/shadow/SliderThumbElement.h: * html/shadow/mac/ImageControlsButtonElementMac.cpp: * html/shadow/mac/ImageControlsRootElementMac.cpp: * rendering/RenderAttachment.cpp: * rendering/RenderAttachment.h: * rendering/RenderBlock.cpp: * rendering/RenderBlock.h: * rendering/RenderBlockFlow.cpp: * rendering/RenderBlockFlow.h: * rendering/RenderBox.cpp: * rendering/RenderBox.h: * rendering/RenderBoxModelObject.cpp: * rendering/RenderBoxModelObject.h: * rendering/RenderButton.cpp: * rendering/RenderButton.h: * rendering/RenderCombineText.cpp: * rendering/RenderCombineText.h: * rendering/RenderCounter.cpp: * rendering/RenderCounter.h: * rendering/RenderDeprecatedFlexibleBox.cpp: * rendering/RenderDeprecatedFlexibleBox.h: * rendering/RenderDetailsMarker.cpp: * rendering/RenderDetailsMarker.h: * rendering/RenderElement.cpp: * rendering/RenderElement.h: * rendering/RenderEmbeddedObject.cpp: * rendering/RenderEmbeddedObject.h: * rendering/RenderFileUploadControl.cpp: * rendering/RenderFileUploadControl.h: * rendering/RenderFlexibleBox.cpp: * rendering/RenderFlexibleBox.h: * rendering/RenderFragmentContainer.cpp: * rendering/RenderFragmentContainer.h: * rendering/RenderFragmentContainerSet.cpp: * rendering/RenderFragmentContainerSet.h: * rendering/RenderFragmentedFlow.cpp: * rendering/RenderFragmentedFlow.h: * rendering/RenderFrameBase.cpp: * rendering/RenderFrameBase.h: * rendering/RenderFrameSet.cpp: * rendering/RenderFrameSet.h: * rendering/RenderFullScreen.cpp: * rendering/RenderFullScreen.h: * rendering/RenderGrid.cpp: * rendering/RenderGrid.h: * rendering/RenderHTMLCanvas.cpp: * rendering/RenderHTMLCanvas.h: * rendering/RenderImage.cpp: * rendering/RenderImage.h: * rendering/RenderImageResourceStyleImage.cpp: * rendering/RenderImageResourceStyleImage.h: * rendering/RenderInline.cpp: * rendering/RenderInline.h: * rendering/RenderLayerModelObject.cpp: * rendering/RenderLayerModelObject.h: * rendering/RenderLineBreak.cpp: * rendering/RenderLineBreak.h: * rendering/RenderListBox.cpp: * rendering/RenderListBox.h: * rendering/RenderListItem.cpp: * rendering/RenderListItem.h: * rendering/RenderListMarker.cpp: * rendering/RenderListMarker.h: * rendering/RenderMedia.cpp: * rendering/RenderMedia.h: * rendering/RenderMediaControlElements.cpp: * rendering/RenderMediaControlElements.h: * rendering/RenderMenuList.cpp: * rendering/RenderMenuList.h: * rendering/RenderMeter.cpp: * rendering/RenderMeter.h: * rendering/RenderMultiColumnFlow.cpp: * rendering/RenderMultiColumnFlow.h: * rendering/RenderMultiColumnSet.cpp: * rendering/RenderMultiColumnSet.h: * rendering/RenderMultiColumnSpannerPlaceholder.cpp: * rendering/RenderMultiColumnSpannerPlaceholder.h: * rendering/RenderObject.cpp: * rendering/RenderObject.h: * rendering/RenderProgress.cpp: * rendering/RenderProgress.h: * rendering/RenderQuote.cpp: * rendering/RenderQuote.h: * rendering/RenderReplaced.cpp: * rendering/RenderReplaced.h: * rendering/RenderReplica.cpp: * rendering/RenderReplica.h: * rendering/RenderRuby.cpp: * rendering/RenderRuby.h: * rendering/RenderRubyBase.cpp: * rendering/RenderRubyBase.h: * rendering/RenderRubyRun.cpp: * rendering/RenderRubyRun.h: * rendering/RenderRubyText.cpp: * rendering/RenderRubyText.h: * rendering/RenderScrollbarPart.cpp: * rendering/RenderScrollbarPart.h: * rendering/RenderSearchField.cpp: * rendering/RenderSearchField.h: * rendering/RenderSlider.cpp: * rendering/RenderSlider.h: * rendering/RenderTable.cpp: * rendering/RenderTable.h: * rendering/RenderTableCaption.cpp: * rendering/RenderTableCaption.h: * rendering/RenderTableCell.cpp: * rendering/RenderTableCell.h: * rendering/RenderTableCol.cpp: * rendering/RenderTableCol.h: * rendering/RenderTableRow.cpp: * rendering/RenderTableRow.h: * rendering/RenderTableSection.cpp: * rendering/RenderTableSection.h: * rendering/RenderText.cpp: * rendering/RenderText.h: * rendering/RenderTextControl.cpp: * rendering/RenderTextControl.h: * rendering/RenderTextControlMultiLine.cpp: * rendering/RenderTextControlMultiLine.h: * rendering/RenderTextControlSingleLine.cpp: * rendering/RenderTextControlSingleLine.h: * rendering/RenderTextFragment.cpp: * rendering/RenderTextFragment.h: * rendering/RenderVTTCue.cpp: * rendering/RenderVTTCue.h: * rendering/RenderVideo.cpp: * rendering/RenderVideo.h: * rendering/RenderView.cpp: * rendering/RenderView.h: * rendering/RenderWidget.cpp: * rendering/RenderWidget.h: * rendering/mathml/RenderMathMLBlock.cpp: * rendering/mathml/RenderMathMLBlock.h: * rendering/mathml/RenderMathMLFenced.cpp: * rendering/mathml/RenderMathMLFenced.h: * rendering/mathml/RenderMathMLFencedOperator.cpp: * rendering/mathml/RenderMathMLFencedOperator.h: * rendering/mathml/RenderMathMLFraction.cpp: * rendering/mathml/RenderMathMLFraction.h: * rendering/mathml/RenderMathMLMath.cpp: * rendering/mathml/RenderMathMLMath.h: * rendering/mathml/RenderMathMLMenclose.cpp: * rendering/mathml/RenderMathMLMenclose.h: * rendering/mathml/RenderMathMLOperator.cpp: * rendering/mathml/RenderMathMLOperator.h: * rendering/mathml/RenderMathMLPadded.cpp: * rendering/mathml/RenderMathMLPadded.h: * rendering/mathml/RenderMathMLRoot.cpp: * rendering/mathml/RenderMathMLRoot.h: * rendering/mathml/RenderMathMLRow.cpp: * rendering/mathml/RenderMathMLRow.h: * rendering/mathml/RenderMathMLScripts.cpp: * rendering/mathml/RenderMathMLScripts.h: * rendering/mathml/RenderMathMLSpace.cpp: * rendering/mathml/RenderMathMLSpace.h: * rendering/mathml/RenderMathMLToken.cpp: * rendering/mathml/RenderMathMLToken.h: * rendering/mathml/RenderMathMLUnderOver.cpp: * rendering/mathml/RenderMathMLUnderOver.h: * rendering/svg/RenderSVGBlock.cpp: * rendering/svg/RenderSVGBlock.h: * rendering/svg/RenderSVGContainer.cpp: * rendering/svg/RenderSVGContainer.h: * rendering/svg/RenderSVGEllipse.cpp: * rendering/svg/RenderSVGEllipse.h: * rendering/svg/RenderSVGForeignObject.cpp: * rendering/svg/RenderSVGForeignObject.h: * rendering/svg/RenderSVGGradientStop.cpp: * rendering/svg/RenderSVGGradientStop.h: * rendering/svg/RenderSVGHiddenContainer.cpp: * rendering/svg/RenderSVGHiddenContainer.h: * rendering/svg/RenderSVGImage.cpp: * rendering/svg/RenderSVGImage.h: * rendering/svg/RenderSVGInline.cpp: * rendering/svg/RenderSVGInline.h: * rendering/svg/RenderSVGInlineText.cpp: * rendering/svg/RenderSVGInlineText.h: * rendering/svg/RenderSVGModelObject.cpp: * rendering/svg/RenderSVGModelObject.h: * rendering/svg/RenderSVGPath.cpp: * rendering/svg/RenderSVGPath.h: * rendering/svg/RenderSVGRect.cpp: * rendering/svg/RenderSVGRect.h: * rendering/svg/RenderSVGResourceClipper.cpp: * rendering/svg/RenderSVGResourceClipper.h: * rendering/svg/RenderSVGResourceContainer.cpp: * rendering/svg/RenderSVGResourceContainer.h: * rendering/svg/RenderSVGResourceFilter.cpp: * rendering/svg/RenderSVGResourceFilter.h: * rendering/svg/RenderSVGResourceFilterPrimitive.cpp: * rendering/svg/RenderSVGResourceFilterPrimitive.h: * rendering/svg/RenderSVGResourceGradient.cpp: * rendering/svg/RenderSVGResourceGradient.h: * rendering/svg/RenderSVGResourceLinearGradient.cpp: * rendering/svg/RenderSVGResourceLinearGradient.h: * rendering/svg/RenderSVGResourceMarker.cpp: * rendering/svg/RenderSVGResourceMarker.h: * rendering/svg/RenderSVGResourceMasker.cpp: * rendering/svg/RenderSVGResourceMasker.h: * rendering/svg/RenderSVGResourcePattern.cpp: * rendering/svg/RenderSVGResourcePattern.h: * rendering/svg/RenderSVGResourceRadialGradient.cpp: * rendering/svg/RenderSVGResourceRadialGradient.h: * rendering/svg/RenderSVGRoot.cpp: * rendering/svg/RenderSVGRoot.h: * rendering/svg/RenderSVGShape.cpp: * rendering/svg/RenderSVGShape.h: * rendering/svg/RenderSVGTSpan.cpp: Added. * rendering/svg/RenderSVGTSpan.h: * rendering/svg/RenderSVGText.cpp: * rendering/svg/RenderSVGText.h: * rendering/svg/RenderSVGTextPath.cpp: * rendering/svg/RenderSVGTextPath.h: * rendering/svg/RenderSVGTransformableContainer.cpp: * rendering/svg/RenderSVGTransformableContainer.h: * rendering/svg/RenderSVGViewportContainer.cpp: * rendering/svg/RenderSVGViewportContainer.h: Source/WTF: This just adds an easy way of using the bmalloc IsoHeap API in WebKit. If bmalloc is not enabled, these macros will just make the object FastMalloced. * WTF.xcodeproj/project.pbxproj: * wtf/FastTLS.h: * wtf/IsoMalloc.h: Added. * wtf/IsoMallocInlines.h: Added. Canonical link: https://commits.webkit.org/195445@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@224537 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-11-07 19:21:52 +00:00
void scavenge();
private:
Mutex* m_lock;
FixedVector<void*, logCapacity> m_objectLog;
};
} // namespace bmalloc